Version 1.2 (version notes at bottom of file; major news in this document are labeled with «LBnew» or similar)
A pair of programs that diagnose Nota Bene Ibid's data files. IBIDIAG.RUN is an XPL program that runs from inside Nota Bene. The main job is done by IBIDIAG.EXE, which IBIDIAG.RUN automatically calls. IBIDIAG.EXE does three things: analyzes the use of record numbers, makes notes of several suspect conditions, and writes copies of multiple-occurrence records (if any are found) to a file. IBIDIAG.RUN then checks those multiple occurrences, and finally it calls the report file to the screen.
«MDUL»WHY?«MDNM»
Ibid's indexing and maintenance functions are known to malfunction if there is something disorderly about the data files. In worst cases, Maintain erases records permanently. Fateful conditions arise easily when data files are edited manually, but it is possible to introduce problems even via Ibid's editing template. IBIDIAG helps to detect and correct those problems.
«MDUL»WHEN?«MDNM»
■ «MDBU»Before and after running Maintain in Ibid«MDNM». Save the report file produced before you run Maintain under another name«FN1«RP»The report remains in its window, but saving to a file under another name makes things safer. », run Maintain, run IBIDIAG again, compare the new report with the Before report to see that Maintain has deleted all duplicates and all records marked 'deleted'«FN1«RP»These are records whose last occurrence only contains the record number field. Note, however, that if you reuse records marked deleted, IBIDIAG will list them as 'marked deleted'. », and not made any unwanted deletions. If the record number use does not check out good, go back to your backups, edit your NBBIB*.DAT files manually to get rid of (the most suspicious of) the conditions reported by IBIDIAG, and see if this makes a difference in running Maintain.
NB: Before running Maintain, be sure that you either have a backup set of the current NBBIB*.DAT files or that the NBBIB*.DBK slot is vacant, i.e., that there are no NBIBB*.DBK files in the current Ibid directory. Despite what Ibid's screen leads one to believe, Maintain will NOT make backups of the current data files if earlier backups remain in the Ibid directory!
■ «MDBU»Any time you notice or suspect problems in your Ibid data files«MDNM»: Ibid's Lookup or Select listings claim you have bad records, or you don't find items that should be there, or you have entered new items on several computers, or have added records using both Ibid's edit template and direct editing of *.DAT files, and are not sure whether new records in NBBIB.DAT have record numbers that were in fact already in use.
«MDUL»HOW?«MDBU»«MDNM»
Run IBIDIAG, study the results, edit the NBBIB*.DAT files as you see prudent.
IBIDIAG only examines, it does not try to fix anything.«FN1«RP»You may use other programs for specific tasks, such as Bob Webb's XPL program for checking and adding field-end dots, if there are lots of those lacking; in a large data file with just one or two missing dots, 'manual' editing may be faster, though.»
«MDBU»Running from NB:
«MDNM»
Normally, IBIDIAG would be run from inside NB.
Place copies of IBIDIAG.EXE and IBIDIAG.RUN to directories that are mentioned in DOS's Path. Inside NB, go to the Action/Command line, type:
«MDBO»RUN IBIDIAG.RUN<F10/Return>«MDUL»«MDNM»
If you have a sluggish computer and lots of entries that have been edited, you may want to run the program without the slower XPL part that compares duplicate entries. If so, add the parameter 'N':«FN1«RP»This only skips the comparison part done by the XPL program. IBIDIAG.EXE still writes copies of duplicates, if found, to IBIDIAG.DBL. This is pretty fast, though, so I did not bother to provide a switch for turning it off, but may do so if there is enough need for it.»
«MDBO»RUN IBIDIAG.RUN,N<F10/Return>«MDUL»«MDNM»
Note: You can run IBIDIAG.RUN from any directory, but IBIDIAG.RUN expects that you have selected an IBID database (or have one by default), otherwise it will look for NBBIB*.DAT files in the current directory.«FN1«RP»However, if you use the new `-255' switch (see below), an Ibid directory must be defined, either by default or explicit selection.»
«LBnew in 1.2»To make IBIDIAG skip reporting occurrences of ASCII 255, use the switch -255:
«MDBO»RUN IBIDIAG.RUN,-255<F10/Return>«MDNM»
Memory requirements: I have been able to run IBIDIAG.RUN+EXE from as little as 69 K of free memory (shown by F7 in NB), but I don't know how much memory it actually requires. If it does not seem to run properly, try to increase memory by unloading files or Ibid or Orbis or something before running it again.
Note: If IBIDIAG.RUN cannot find IBIDIAG.EXE (because it has been deleted or renamed, or DOS's path no longer includes its location, say), you might still get a report file, IBIDIAG.REP, to the screen, but most likely it will be from a previous run. When in doubt, check the date and time given at top of the report.
«MDBU»Running from DOS:
«MDNM»
IBIDIAG.EXE can be run from the DOS prompt:
«MDBO»«MDBO»IBIDIAG <IBIDIR>«MDUL»«MDNM» [-255]
«MDUL»«MDNM»
Where IBIDIR is the NBBIB*.DAT files directory; if the directory parameter is lacking, IBIDIAG.EXE will check current directory. The optional parameter `-255' turns off reporting of ASCII 255 characters.«FN1«RP»If you use the '-255' switch, also the directory must be explicitly given.
»
Note: Automatic duplicate comparison works only from inside NB. IBIDIAG.EXE writes a main report file, IBIDIAG.REP, and a listing of multiple-occurrence records (if found) to another file, IBIDIAG.DBL.
«MDBU»A note on filenames«MDNM»
IBIDIAG should not do anything to your files, but it uses these filenames for its reports and intermediate management:
IBIDIAG.REP
IBIDIAG.DBL records whose numbers occur more than once
IBIDIAG.SOR same sorted by rec#
Any existing files with these names in the Ibid directory will be overwritten without warning. If you want to save report files (such as for a Before-and-After Maintain check), save them under other filenames.
«MDUL»What does IBIDIAG report?«MDNM» (and what to do about the things reported)
«TS6,10,15»«IP0,#1»«DC1=1.1»
«C1» The existence of «MDBU»NBBIB*.DBK«MDNM» files in the examined directory
These should be moved or renamed before running Maintain. Otherwise Maintain will not save the current NBBIB*.DAT files.
«C1» «MDBU»Record number use«MDNM»
«C2» The «MDBU»range«MDNM» of record numbers used
«C2» List of «MDBU»unused«MDNM» numbers, their count
Check this before and after running Maintain. If the count increases, rescue the (unacceptably) deleted items from your backups.
«C2» List of numbers used «MDBU»more than once«MDNM», their count
Normally, these would indicate records that have been edited since last Maintain. However, Maintain sometimes makes more duplicates instead of weeding all out. It is also possible that different items have received the same record number, in which case the next Maintain would permanently erase a unique item, not just a dated version of one. Note: in a normal case, a record number that is marked as deleted appears more than once (first as a real entry, then as an `empty' record number signaling deletion) and IBIDIAG lists it both as number `used more than once' and as a number `marked deleted'.
«C2» List of numbers «MDBU»marked deleted«MDNM», their count
You may want to check that you really want to delete these. Note: IBIDIAG lists here any record number that is marked as deleted (i.e., has no contents but the record number field«FN1«RP»IBIDIAG does nothing with NBBIB.IDX or NBBIB.REC. "Marked deleted" is a record in a NBBIB*.DAT file that looks like this
R#:11·
AEPG AF
Mere reindexing does not remove these from the data files. Only Maintain does.
Once IBIDIAG sees this record, it flags rec no 11 as deleted, and does not bother to change the flag even if there were a later R#:11 with real contents. I assumed this would be rare case, if one uses Maintain at least sometimes (then these record numbers would be just unused).
»), even if the number were reused later in the NBBIB*.DAT files.«FN1«RP»After running Maintain the number of records marked deleted should be zero, while the number of unused record numbers should be what it was before Maintain plus those previously marked for removal.»
«C1». Several «MDBU»conditions known or suspected to cause problems«MDNM»
These are listed in the order they are found when scanning the files NBBIB000.DAT through NBBIB[HIGHEST#].DAT + NBBIB.DAT. The listing gives a label of the condition and its location both as Fileposition and Record number. You can then call the affected file(s) to the screen and find the interesting spots either by JuMPing to the position or searching for the record number.
«LBnew remark»If the fileposition indicators seem starkly inaccurate, the reason may be that you are using Lingua and the program expands one-byte accented characters into 3-byte versions when the file is called to screen (while IBIDIAG counted them as one byte). You get accurate readings by suppressing the expansion. Call the file with the '/x' switch (ca /x filename) or suppress expansion for .DAT files (F1->Set->Other->File->Internal), or switch to non-Lingua mode. Often, though, you would find the spot by searching for the record number given.
Another reason for a (usually milder) gap that seems to develop between the filepositions listed by IBIDIAG and the actual symptom spots is that your correcting the problems usually affects the number of bytes in the file. So if you have a lot to edit, better JuMP first to the «MDIT»highest«MDNM» spot mentioned, then the next highest, etc.
«C2» Field-end dot missing
Flagged as `Odd field/record end'
More exactly, the program warns of every case where a paragraph marker is preceded by anything but a field-end dot («MDBI»·«MDNM») or a record-end («MDBI» AEPG AF«MDNM»)
Serious. The dot must be there.
«C2» «MDBI»R#:«MDNM» not preceded by a «MDBI» AEPG AF«MDNM» + «MDBI»CR«MDNM»
Flagged as '`R#:' in odd spot'
There is a 'R#:' sequence but it is not at the beginning of line.
Probably very serious. I would avoid these altogether.
«C2» Record number not followed by a field-end dot (e.g., «MDBI»R#:23 ·«MDNM»)
Maybe serious. Delete the extraneous character.
«C2» Single Line Feed (i.e., no Carriage Return before it)
Maybe serious. Can crop up when downloading data from sources that use a single LF to mark end of line: Unix, even some DOS & Windows wordproscessors). Visible only in NB's expanded mode. Should be deleted or replaced by a regular NB paragraph marker, if it is at a field end.«FN1«RP»If there are many of these, one could use a program to change them into regular para markers. »
«C2» Single Carriage Return (i.e., no Line Feed after it)
Maybe serious. Sources usually non-NB. In NB looks like a regular para marker, but behaves differently.
«LBnew in 1.2»Flagged now as `Odd Carriage Return'
«C2» Opening command bracket following a colon (field «MDBI»ID: AE«MDNM»)
A usual source for this would be a print mode delta.
Removal of this condition is known to have prevented problems in some cases, but there have also been reports of happy coexistence, and NB 4.2 may be more tolerant of this condition than earlier version.
Known to cause problems (unwanted deletions or duplicates«FN1«RP»I witnessed a DELETION under NB 4.1 and a great number of extraneous duplicates under 4.2») in Maintain in some cases. See preceding point.
«C2» Any ASCII 255 character
Usually these indicate the presence of a NB three-byte character; these can be benign, such as En and Em dash probably are, but also unwanted, such as Lingua 3-byte versions of characters that have a standard one-byte representation. The program currently lists all appearances of 255. Some 255 characters at least in field-beginning positions seem to have been involved in record loss (when running Maintain), but it may be that mid-field positions are not that risky.
«LBnew in 1.2»For turning off the 255 check, see bottom of this document.
«LBnew note Jan 18, 1995, at 21:50»Recently Maintain produced duplicate copies of some half a dozen records. I then deleted to ' ─ ' dashes from a record added since last maintain. After that Maintain would run flawlessly.
«C2» Tilde
Conditional hyphen, not visible in Ibid's views.
Known to cause problems at least if it follows immediately the field ID colon; the program presently warns of all tildes irrespective of their position: I've decided not to use them.
«C1». Comparison of contents of «MDBU»multiple appearances under similar record numbers«MDNM».
If you add records to Ibid on more than one computer, or even on one computer add records both directly to NBBIB.DAT and via Ibid's editing template, it can easily happen that different items are given the same R#. IBIDIAG runs a rough check to detect such cases.
IBIDIAG.EXE writes to IBIDIAG.DBL copies of all instances of any record whose number occurs more than once. IBIDIAG.RUN sorts that file (on record number, result file is IBIDIAG.SOR) and runs a simple comparison of the first substantive lines of adjacent records with the same R#.
Limitations: Comparison is based on the first field starting with a two-letter ID, usually the AUthor. Hence, there may be false alerts, say, for a mere change of punctuation. And «MDBU»false non-alerts«MDNM»: e.g., different titles by the same author. A further restriction is that this part of the program cannot tell records marked for deletion from other cases where the record number appears more than once; so there may be additional noise caused by items marked for deletion.
«MDUL»About«MDNM»
«IP»
Written by Jukka-Pekka Takala, Jukka-Pekka.Takala@Helsinki.Fi
October-November 1994
«MDUL»Version 1.«MDNM»
This program may be freely used for non-commercial purposes. It may be distributed freely, provided that all three files IBIDIAG.EXE, IBIDIAG.RUN, and IBIDIAG.DOC are included. You may customize IBIDIAG.RUN for your purposes, and also give away copies of your customized version as long as you (i) indicate that that is what it is and (ii) provide a copy of the original IBIDIAG.RUN as well. About any other use, please, contact me.
Suggestions and bug reports welcome.
This is a combination and expansion of two of my earlier programs for Ibid security.
The first one, IB_2CHK.RUN & .EXE, I wrote in 1993 to check and compare duplicate entries, after having realized that Maintain would happily erase «MDBU»R#:1, Smith«MDNM» if a later «MDBU»R#:1, Jones«MDNM» existed. Have saved several tomes since. IB_2CHK was uploaded in NB4UTS02.ZIP in Fall 1993 but dropped from the next version of the archive.
The other program, IB_NOS.RUN & .EXE, I wrote in early 1994 to report on unused record numbers in Ibid's data files. I believe this was after reports on the NB list that Maintain sometimes seems to erase even the last or only occurrence of a record. This program was never uploaded to an archive, but I mentioned it on the Nota Bene list (June 1994) and sent copies to a few interested persons.
IBIDIAG does both of those jobs, and faster.
In addition, I decided to add routines for checking various conditions whose fixing might prevent some of the problems. Most of the problems can be detected by viewing (in the expanded mode) NBBIB*.DATs as regular files, but this program does routine checking faster. It also does it faster than some existing XPL utilities do.
The symptoms that the program now checks were gathered mostly from ideas presented on the Nota Bene list. I can recall that I have picked up ideas of conditions worth checking from entries by Bob Webb, John Kloppenborg, Mervyn Bennun, and Peter Binkley but there have certainly been others.
It seems that NB 4.2 has not made this program superfluous.
«MDUL»Version 1.01«MDNM» November 19, 1994
Version 1 sometimes failed to open all the NBBIB*.DAT files, particularly if *.DBK files were present and there were more than two data files. I never noticed the problem myself, but after Phil Glaser's and Jerry Walsh's reports on the problem I was able to reproduce it (after creating a number of bogus data files in addition to my two real ones). I hope that the explicit Closing of all input files which I now added to IBIDIAG.EXE will do away with the problem, it seemed to do so for me.
I also added a few more thoughts and guesses on the nature and fixing of several conditions + fixed a few typos.
Thanks to Phil Glaser and Jerry Walsh for their comments and suggestions.
«MDUL»Version 1.2«MDNM» December 10-21, 1994
Fixed an omission in the code that let single Carriage Returns go undetected if they followed a valid record end marker.
A single Carriage Return is now reported with a label of its own (`Odd Carriage Return'); in earlier versions it was checked together with the presence of a field-end dot or a record-end AEPG AF in the end of the previous line and remarked as 'Odd field/record end'. (Which made it difficult to spot in NB, because a single Carriage Return looks just like a regular CR-LF pair, or a para marker.) Thanks to Guido Milanese whose report on single CR problems made me recognize these problems.
As many users have lots of ASCII 255 characters that seem to do no harm, I added a switch for turning off reports of 255s.
Changed the way IBIDIAG.RUN compares duplicates. In the previous versions a record marked for deletion put comparison out of sync. Now it works more reliably also in those cases.
